Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach

In this paper we address the problem of decision making within a Markov de-cision process (MDP) framework where risk and modeling errors are taken intoaccount. Our approach is to minimize a risk-sensitive conditional-value-at-risk(CVaR) objective, as opposed to a standard risk-neutral expectation. We refer tosuch problem as CVaR MDP. Our first contribution is to show that a CVaR objec-tive, besides capturing risk sensitivity, has an alternative interpretation as expectedcost under worst-case modeling errors, for a given error budget. This result, whichis of independent interest, motivates CVaR MDPs as a unifying framework forrisk-sensitive and robust decision making. Our second contribution is to presentan approximate value-iteration algorithm for CVaR MDPs and analyze its conver-gence rate. To our knowledge, this is the first solution algorithm for CVaR MDPsthat enjoys error guarantees. Finally, we present results from numerical experi-ments that corroborate our theoretical findings and show the practicality of ourapproach.